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Editorial  Guidelines 


With  this  issue,  the  Cornbelt  Education  Review  is  enjoying  a 
modest  beginning,  with  its  small  selection  of  articles  and 
limited  circulation.   As  a  graduate  "journal,  it  is  intended 
to  provide  an  opportunity  for  professional  sharing  of  the  re- 
search efforts  and  concerns  of  beginning  researchers.   We  are 
endeavoring  to  encourage  the  preparation  of  publications,  the 
flexing  of  editorial  skills,  and  the  use  of  research  methodo- 
logies which  may  not  be  completely  developed  or  even  widely 
accepted  within  the  academic  community  as  a  whole, 
naturalistic  or  experimental  research,  book  reviews,  evaluations, 
The  Cornbelt  Education  Review  offers  graduate  students  space 
to  explore  research  methodologies  alternative  to  those  perhaps 
required  in  the  dissertation.   More  importantly,  these  attempts 
to  translate  theory  into  practice  help  underscore  some  of  the 
prominant  educational  issues  which  are  currently  under  discus- 
sion in  the  College  and  beyond. 

Editorial  Guidelines  for  the  journal  suggest  the  inclusion  of 

or  surveys  by  graduate  students  from  Elementary  and  Early  Child- 
hood Education.   Articles  of  approximately  2000-4000  words  in 
APA  style  should  be  submitted  to  the  Editorial  Board,  330 
Education  Building  for  refereed  selection. 

We  wish  to  acknowledge  and  express  appreciation  for  the  efforts 
of  Dean  J..  Myron  Atkin,  Department  Chairman  Harold  Lerch  and 
the  journal  Advisor  Bernard  Spodek,  who  were  instrumental  in 
initiating  its  conceptualization  and  supported  its  development.. 


The  Editorial  Staff 

Judith  Pickle 
Elizabeth  Rosen 
Patricia  Scheyer 
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Congratulations  to  the  Department  of  Elementary  and  Early 
Childhood  Education  for  launching  the  Corn  Belt  Education  Review. 
Congratulations  are  in  order  not  so  much  because  the  world  needs 
another  education  journal  but  because  the  voices  of  a  strong  and 
influential  Department  just  might  have  an  impact  at  a  time  when 
its  field  is  beset  by  confusion  and  uncertainty.   The  Department 
of  Elementary  and  Early  Childhood  Education  at  the  University  of 
Illinois  at  Urbana-Champaign  has  a  distinguished  history;   it  is 
staffed  currently  by  an  extraordinarily  strong  faculty;   it  con- 
tinues to  attract  some  of  the  most  able  graduate  students  in  the 
College.   Perhaps  this  group  can  speak  to  issues  associated  with 
accountability  in  education,  with  the  state  of  our  teacher  edu- 
cation programs  across  the  country,  with  the  challenges  of  con- 
tinuing education  for  classroom  teachers,  with  racial  conflict 
and  bilingualism  in  the  schools,  with  curriculum  questions,  and 
with  the  highly-charged  issue  of  selection  for  the  teaching  pro- 
fession. 

In  most  departments  of  elementary  and  early  childhood  edu- 
cation, the  faculty  and  students  are  preoccupied  with  declining 
enrollments  and  budgets.   Fortunately,  fifteen  years  ago,  this 
Department  decided  to  limit  undergraduate  teacher  education  num- 
bers at  the  University  of  Illinois  at  Urbana-Champaign  and  focus 
on  graduate-level  training,  research,  and  development.   Partly 
as  a  result,  the  Department  is  probably  in  as  strong  a  position 
programmatically  as  any,  anywhere. 

For  all  these  reasons  the  initiation  of  the  Corn  Belt 
Education  Review  is  an  event  that  offers  promise  for  according 
thoughtful  people  the  opportunity  to  reflect  on  current  problems 
from  both  a  privileged  and  a  distinguished  platform.   I  look 
forward  to  seeing  the  results. 


J.  Myron  Atkin,  Dean 
April  1977 
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PROTOCOLS— A  REVIEW  SYNTHESIS 

Nancy  Blair 
Elementary  Education 


Introduction 


B.  0.  Smith's  Teachers  for  the  Real  World  was  the  cata- 
lyst for  the  protocol  materials  development  movement.   Smith 
voiced  a  concern  for  the  failure  of  teacher  training  institu- 
tions to  prepare  teachers  to  analyze  new  situations  against 
a  background  of  relevant  theory.   Smith  states  that  teachers, 
"typically. . .base  their  interpretations  of  behavior  on  intui- 
tion and  common  sense... If  the  teacher  is  incapable  of  under- 
standing classroom  situations  the  actions  he  takes  will 
often  increase  his  difficulties."   (Smith,  1969) 

Protocols  are  original  records  of  events  of  educational 
significance.   They  are  utilized  to  permit  teachers  or  pro- 
spective teachers  to  interpret  events  with  the  aid  of  appro- 
priate concepts  from  related  fields  of  study.   (Cruikshank, 
1974)   They  are  reproductions  of  student,  teacher  or  parent 
behavior  recorded  on  film,  cassette  or  filmstrips.   They 
exemplify  educational  concepts  and  are  designed  to  teach 
the  application  of  these  concepts  and  to  train  school  person- 
ell  to  analyze  and  classify  behavior. 

Protocols  offer  several  advantages  over  direct  observa- 
tion in  real  classrooms.   In  real  life,  behaviors  are  fleet- 
ing.  They  may  be  missed  or  they  may  be  recalled  faultily 
for  later  analysis.   With  the  use  of  protocols  a  behavior 
can  be  captured  and  replayed  as  many  times  as  necessary  for 
study  and  analysis.   Thus,  protocols  are  more  efficient  for 
analysis  since  particular  behaviors  and  related  concepts  can 
be  utilized  at  appropriate  times  in  the  students  training. 
(Gunderson,  1974)   On  the  other  hand,  protocols  lack  the 
spontaneity  and  some  of  the  "realness"  of  direct  observation. 
Ideally,  protocols  would  provide  examples  of  behavior  that 
would  help  the  observer  acquire  the  conceptual  knowledge 
necessary  to  interpret  pedagogical  events. 

Claims 

Numerous  claims  have  been  made  as  to  the  value  of 
protocol  materials.   Orlosky's  support  for  the  use  of  proto- 
cols is  based  on  the  belief  that  "if  behaviors  could  be 
captured  on  film,  students  in  training  could  see  the  behav- 
ior exemplifying  a  given  concept  and  be  better  prepared  to 
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recognize  and  classify  the  behavior."   (Orlosky,  1974) 
Gunderson  asserts  that  certain  cognitions  come  from  the 
exposure  to  protocols.   She  feels  they  teach  concepts  and 
principles  used  in  interpreting  behavior,  and  that  they  teach 
knowledge  about  knowledge  as  well  as  self-understanding. 
(Gunderson,  1974)   Gee  and  Berliner  have  broken  down  the 
acquisition  of  concepts  to  interpret  behavior  into  three 
stages:   1)  functional  knowledge  of  psychology,  philosophy 
and  social  concepts  relevant  to  teaching,  2)  ability  to  inter- 
pret behavioral  situations  in  terms  of  these  concepts  and 
3)  the  ability  to  formulate  plans  for  teaching  based  on  the 
first  two  skills.   In  other  words  protocols  give  teachers 
a  basis  for  decision  making  in  the  classroom.   (Gee  and 
Berliner,  1971) 

Smith  fears  that  there  is  a  danger  of  reducing  teacher 
training  to  the  development  of  teaching  skills  alone.   He 
claims  that  protocols  can  bridge  the  gap  between  theory  and 
practice.   He  emphasizes  interpretive  and  replicative  uses  of 
theoretical  knowledge  and  the  use  of  protocols  to  develop  a 
teacher's  conceptual  system — using  the  theoretical  knowledge 
in  real  situations.   He  states: 

By  being  involved  repeatedly  in  the  process  of 
analyzing  and  interpreting  them  (behavioral 
situations) ,  the  prospective  teacher  will  learn 
to  interpret  quickly  and  thoroughly  the  events 
and  episodes  that  happen  as  he  teaches. 
(Smith,  1969) 

Beyond  this,  the  student  will  also  become  aware  of  the  great 
variety  of  ways  of  handling  each  teaching  task  as  well  as 
the  different  ways  teaching  behaviors  and  situations  may  be 
interpreted.   Smith  further  claims  that  through  the  use  of 
protocols,  teachers  will  acquire  an  increased  interest  in 
theory  and  recognize  its  usefulness. 

Selection  of  Concepts 

Developers  of  protocols  have  tried  to  identify  teach- 
ing concepts  and  then  depict  these  concepts  in  protocol 
format.   There  are  several  problems  attached  to  this  approach 
to  concept  selection.   First,  the  concepts  in  the  founda- 
tional areas  of  education  are  weak  due  to  our  lack  of 
knowledge  about  program  outcomes.   (Hudgins,  1972)   Second, 
if  protocols  are  to  be  selected  to  reflect  concepts  with 
generality,  utility,  and  interpretive  possibilities,  we  must 
first  carefully  analyze  the  concepts  identified  in  depth. 
The  dimensions  and  standards  of  such  an  analytic  process  are 
not  widely  understood  in  psychology,  sociology  and 
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anthropology.   (Gleissman,  1972)   Furthermore,  teacher  educa- 
tion is  a  difficult  substantive  area  due  to  the  lack  of  a 
match  between  concepts  and  the  exemplary  behaviors.   In  spite 
of  these  problems  it  was  believed  that  delay  in  production  of 
protocol  materials  pending  definitive  statements  concerning 
concept  selection  would  be  counterproductive. 

Before  production  began  the  concepts  chosen  had  to  be 
clearly  and  precisely  defined.   The  Leadership  Training 
Institute  on  Protocol  Materials  recommended  that  concepts  be 
expressed  in  one  of  four  possible  forms;  classification, 
equivalent  expression,  open  context  and  conditional.   Clas- 
sification calls  for  the  association  of  a  concept  with  a 
category  and  discriminating  criteria  distinguishing  one 
concept  from  another.   In  the  equivalent  expression  form  a 
concept  is  explained  by  a  statement  equivalent  to  word  or 
words  used  to  name  that  concept.   Open  context  is  used  for 
concepts  that  are  not  clearly  definable,  such  as  happiness. 
Conditional  concepts  are  those  that  can  be  explained  only 
in  terms  of  a  context  or  conditions  surrounding  the  event. 
(Smith  and  Orlosky,  1973) 

Initially,  all  of  the  developers  listed  the  concepts  that 
they  planned  to  produce  in  protocol  form.   As  work  progressed 
the  initial  choices  were  generally  maintained  but  with  great- 
er emphasis  on  clarification  and  supporting  information.   The 
developers  eventually  identified  concepts  that  are  part  of  a 
system  rather  than  isolated  from  each  other. 

Building  the  Concept 

After  a  concept  has  been  identified,  it  must  be  analyzed 
and  evaluated.   The  following  steps  in  evaluating  a  concept 
have  been  used. 

1.  Is  it  representable? 

2.  Is  it  significant?   (high  utility,  critical  to 
teacher  education,  and  shows  empirical  substantiation) 

3.  What  are  the  learning  objectives  related  to  this 
concept? 

4.  Who  is  the  anticipated  user? 

5.  How  will  it  be  field  tested?   (Gunderson,  1974) 


-3- 


In  analyzing  a  concept,  one  must  identify  the  range  and 
types  of  behaviors  to  be  captured  to  portray  the  concept  fairly. 
It  is  necessary  to  specify  the  dimensions  and  values  of  the 
concept  in  order  to  insure  comprehensive  representation.   To 
clearly  define  a  concept  one  must  consider  complexity  of 
examples,  sequencing,  range  of  discriminations  the  learner 
makes,  definitions  and  descriptions.   (Hudgins,  1972) 

The  amount  of  literature  on  concept  acquisition  is  an 
indication  of  the  complexity  of  this  field.   Jerome  Bruner, 
in  A  Study  of  Thinking,  refers  to  concept  attainment  as  the 
"process  of  finding  predictive  defining  attributes  that  dis- 
tinguish exemplars  from  nonexemplars  of  the  class  one  seeks 
to  discriminate."  He  suggests  that  examples  and  counter 
examples  given  to  students  optimize  the  acquisition  of  any 
given  concept  regardless  of  the  type  of  learning  strategy 
employed  by  the  student.   (Bruner,  1956) 

While  Gagne  agrees  that  simpler  concepts  can  be  learned 
through  the  contrast  of  positive  and  negative  examples,  he 
further  suggests  that  more  complex  concepts  (i.e.,  work, 
uncle)  which  are  combinations  of  simpler  concepts  are  concepts 
by  definition.   They  are  learned  via  verbal  statements  that 
provide  cues  to  recall  of  component  concepts  and  their  correct 
ordering.   Gagne  asserts  that  there  is  much  to  be  done  in 
conducting  research  on  the  acqusition  of  these  two  types  of 
concepts,  and  that  in  order  for  the  research  to  be  meaningful 
we  must  distinguish  between  simple  and  more  complex  types 
of  concepts.   (Gagne,  1966) 

Hunt  defines  concept  learning  in  terms  of  learning  a 
rule  that  would  apply  to  any  appropriate  stimulus  that  fits 
the  given  concept  and  this  rule  would  be  uniquely  classifi- 
able.  (Hunt,  1962)   This  differs  from  Piaget's  definition 
of  a  concept  as  an  explanatory  rule  that  need  not  be  a  class- 
ification rule.   (Piaget,  1957)   Thus,  the  issue  is  further 
complicated  by  the  lack  of  agreement  on  terminology.   Ausubel 
believes  that  building  a  clear,  stable  and  unambiguous  mean- 
ing of  the  concept  in  the  student's  cognitive  structure  is 
essential.   (Ausubel,  1968) 

Hudgins  has  tried  to  incorporate  some  of  these  theories 
on  concept  acquisition  for  use  with  protocol  materials.  His 
model  calls  for  concept  presentation  that  increases  stimulus 
complexity. 
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Stage  I  Stage  II 

Example  on  film 

Printed  definition  Low  noise 

Written  examples  Cues,  labels 

Stage  III  Stage  IV 

Positive  and  negative  "Slice  of  life" 

examples  (less  editing) 
Increased  noise 

Fade  cues,  labels  (Hudgins,  1972) 


Another  issue  that  arises  is  whether  or  not  teacher- 
learning  episodes  should  be  staged  for  filming.   This  has 
caused  concern  among  developers  but  it  is  an  empirical  ques- 
tion that  has  not  yet  been  answered.   Certainly  finances  and 
time  constraints  support  the  staging  of  situations. 
(Cruikshank,  1972) 

The  Use  of  the  Materials 

There  has  been  some  disagreement  between  the  heuristic 
and  didactic  use  of  protocols.   Should  students  watch  the 
protocol  material  and  arrive  at  the  concept  inductively  or 
would  direct  guidance  be  more  efficient?   Neither  research 
nor  theory  lends  adequate  support  to  this  distinction.   (Smith, 
Orlosky,  1973)   Perhaps  the  main  consideration  would  be 
whether  or  not  the  concepts  have  been  presented  in  such  a  way 
to  increase  understanding.   One  such  avenue  for  increased 
understanding  of  a  concept  has  been  suggested  by  Innerd 
and  0' Gorman. 

1.  Information — relevant  theory  presented  as  hypothesis 
to  be  tested. 

2.  Protocol  Material — video  of  real  behavior  shown. 

3.  Analysis — viewer  discusses  behavior  in  relation 
to  theory. 

4.  Discussion — does  the  theory  hold  up  or  is  it 
denied?   (Innerd,  0' Gorman,  1970) 

Evaluation 

There  are  several  types  of  evidence  that  have  been 
compiled  to  demonstrate  the  effectiveness  of  protocol 
materials.   Most  of  the  evidence  is  reported  in  terms  of 
concept  acquisition  and  reactions  to  the  materials  by 
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trainees  and  their  instructor.   Most  of  this  evidence  was 
collected  during  field  tests  of  the  products  rather  than 
large-scale  evaluation  efforts. 

Utah  State  University,  Michigan  State  University,  Far 
West  Laboratory,  Indiana  University  and  Southern  Illinois 
University  at  Edwardsville  have  reported  superiority  in 
concept  growth  in  groups  using  protocol  materials  as  compared 
to  groups  using  traditional  lecture-discussion  and  printed 
materials.   Studies  at  Michigan  State  produce  evidence  not 
only  of  concept  acquisition  but  also  on  its  transfer  to 
simulated  teaching  situations.   (Cooper,  1975)   It  has  not 
been  demonstrated  that  this  transfer  occurred  in  a  real 
classroom  situation. 

Instruments  have  been  designed  by  Protocol  Projects 
Materials  directors  to  measure  attitudes  of  students  to  pro- 
tocol materials.   These  instruments  gather  reactions  to 
technical  quality  and  usefulness  of  content  for  improving 
teaching.   These  result  in  value  judgments  which  for  the 
most  part  are  favorable  to  protocol  products.   Indiana 
University  collected  information  on  the  reactions  of  instruc- 
tors.  Six  instructors  from  several  institutions  gave  positive 
responses  to  the  materials'  quality,  appropriateness  of 
content  and  usefulness  in  promoting  intended  concepts. 
(Cooper,  1975) 

Kleuker  measured  the  effectiveness  of  protocol  materials 
and  training  materials  (methodological  and  skill  training) 
together  and  separately  as  related  to  concept  acquisition. 
She,  too,  found  that  students  using  protocol  materials  were 
superior  in  concept  acquisition  to  a  control  group.   She 
also  found  that  students  who  received  both  protocol  and 
skill  training  performed  better  in  concept  recognition  than 
students  who  just  received  protocol  training.   (Kleuker,  1974) 
One  must  ask  whether  the  added  exposure  to  the  learning 
situation  accounted  for  this  increase  in  performance 
rather  than  the  combination  of  the  two  approaches. 

The  research  conducted  thus  far  is  limited  and  not 
directed  at  the  many  claims  advanced  in  support  of  the  use  of 
protocols.   There  are  many  questions  left  unanswered.   Are 
students  trained  via  protocols  better  able  to  formulate 
plans  for  a  learning  situation  based  on  these  concepts?   Do 
these  students  have  increased  self-understanding?  Does  it 
effect  their  performance  as  a  teacher?  Are  these  students 
able  to  quickly  and  thoroughly  interpret  the  complex  be- 
haviors that  occur  in  the  classroom?   As  Smith  claims,  have 
the  students  developed  an  increased  interest  in  theory  due 
to  their  interaction  with  protocols?   Have  the  students 
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developed  an  increased  recognition  of  the  usefulness  of  theory? 
Have  there  been  any  positive  changes  in  pupil  behavior  that 
would  support  the  use  of  protocol  materials?   At  this  stage, 
the  evaluation  efforts  are  not  congruent  with  the  promises 
made  throughout  the  development  process.   Enough  protocols 
have  been  developed  to  provide  the  necessary  materials  for 
large  scale  evaluation  that  addresses  some  of  the  questions 
raised  here. 

One  would  agree  with  Smith  that  teacher  educators  must  go 
beyond  concerns  for  technique  and  skill  acquisition.   We  need 
to  stimulate  the  intellect  of  the  prospective  teacher  and 
instill  in  him  the  dispositions  to  be  analytical,  synthetical, 
reflective  and  curious.   The  protocol  movement  has  been  a 
fascinating  attempt  to  develop  materials  to  do  just  that  and 
future  investigations  should  determine  its  true  merit.   At  the 
very  least,  encouragement  is  needed  for  careful  evaluation 
of  these  materials  that  goes  beyond  concept  acquisition  and 
student  reactions. 
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REFLECTION  IN  STUDENT  TEACHING 

James  L.  Hoot 
Early  Childhood  Education 

In  his  book,  Life  in  Classrooms,  Jackson  described  the 
classroom  as  a  "proverbial  beehive  of  activity"  (Jackson, 
1968).   Within  this  intense  activity,  educational  decisions 
are  made  by  teachers.   At  issue  in  the  present  study  is  how 
such  decisions  are  made,  i.e,  are  they  made  in  a  haphazard, 
thoughtless,  manner — or  are  they  the  outcome  of  a  reflective 
process  which  Dewey  defined  as  "...active,  persistent,  and 
careful  consideration  of  any  belief  or  supposed  form  of  know- 
ledge in  light  of  the  grounds  that  support  it..."   (Dewey, 
1933). 

Accounts  of  teachers  making  decisions  and  implementing 
curricula  with  little  or  no  thought  given  to  assumptions 
underlying  such  implementation  are  not  rare.   From  her  many 
observations  and  discussions  with  teachers  in  elementary 
classrooms,  for  example,  Durkin  concluded  that  a  major  goal 
of  lessons  was  often  "filling  up  a  day."   Moreover,  she 
stated  that  "While  other  (educational)  goals  were  better, 
some  were  still  not  the  kind  that  could  easily  be  defended 
by  educators"   (Durkin,  1975). 

In  addition  to  concern  about  the  assumptions  underlying 
selection  of  curricular  goals,  recent  reports  have  also 
suggested  a  need  for  teachers  to  become  increasingly  reflec- 
tive about  material  selection.   A  study  by  the  Educational 
Products  Information  Exchange  Institute,  for  example, 
indicated  that  "...99  percent  of  roughly  200,000  curriculum 
materials  on  the  market  were  not  being  tested  prior  to 
publication."   Although  these  "untested"  materials  account 
for  less  than  1%  of  the  national  school  budget  (The  1974 
Cost  of  Education  Index) ,  they  often  dictate  a  large  per- 
centage of  instructional  time. 

The  present  study  began  with  the  assumption  that 
teachers  should  not  serve  a  valet  function  in  making  educa- 
tional decisions.   Rather,  choices  made  by  educators  should 
be  supported  by  a  defensible  belief  system  i.e.,  they  should 
be  the  outcome  of  a  reflective  process.   And,  if  a  reflective 
disposition  can  be  seen  as  a  desirable  teacher  characteristic, 


*For  a  complete  account  of  this  study  see:   James  L. 
Hoot,  "Student  Teacher  Reflection  to  Preactive,  Interactive, 
and  Reactive  Teaching  Concerns  in  Reading  and  Math."   Un- 
published Evluation  Report,  University  of  Illinois,  1976. 
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it  follows  that  institutions  purporting  to  prepare  teachers 
should  assume  greater  responsibility  for  strengthening  reflec- 
tive attitudes  in  its  students.   In  this  view,  teacher  educa- 
tion is  seen  as  being  what  Katz  (1974)  calls  "functionally" 
related  to  what  is  useful  in  the  field,  i.e.,  "...what 
trainees  learn  in  the  program  helps  them  become  good  at 
teaching. " 

The  University  of  Illinois  Early  Childhood  Teacher 
Education  Program  is  presently  in  the  process  of  moving 
toward  a  "liberation"  model  which  recognizes  the  importance 
of  responsible  choice  in  educational  decision-making  (Spodek, 
1975,  Becher,  1976).   With  this  emphasis,  we  have  become 
increasingly  concerned  that  students  being  recommended  for 
certification  from  our  program  have  begun  to  develop  a  pro- 
pensity to  reflect  upon  the  things  they  do  and  the  choices 
they  make.   Further,  viewing  our  students  as  reflecting  as 
well  as  assimilating  agents,  we  would  hope  that  the  nature 
of  our  teacher  education  program  had  been  instrumental  in 
enabling  students  "...to  make  personal  judgments  assessing 
values  and  performance  through  self-reflection..." 
(Schickedanz  and  Spodek,  1975) . 

The  present  study  was  an  outgrowth  of  our  ongoing 
program  evaluation  process.   Although  limited  in  its  scope, 
the  study  attempted  to  assess  the  degree  to  which  a  sample 
of  our  student  teachers  would  provide  evidence  of  reflection 
in  planning,  teaching,  and  evaluating  sample  lessons  in  the 
reading  and  math  content  areas. 

Major  questions  guiding  the  present  inquiry  were: 

1.  How  reflective  will  student  teachers  be  in 
responding  to  interview  items  related  to  selected 
dimensions  of  the  teaching  process  in  the  content 
areas  of  reading  and  math? 

2.  Will  reflection  scores  differ  between  the 
two  content  areas? 

3.  Will  reflection  scores  differ  in  content  areas 
between  the  planning,  teaching  and  evaluating 
aspects  of  teaching? 

4.  What  factors  might  account  for  differences  in 
reflection  scores? 
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Procedure 

In  October,  1975,  five  University  of  Illinois  students 
were  randomly  chosen  from  an  eleven  member  section  of  Elemen- 
tary Education  232  (Supervised  Student  Teaching).   Students 
were  seniors  majoring  in  Early  Childhood  Education  and  all 
were  completing  their  final  year  of  preservice  training. 

While  student  teaching  in  the  Champa ign-Urbana  school 
systems,  subjects  were  asked  to  participate  in  two  interview 
sessions  which  were  to  follow  formal  observations  of  a  reading 
and  a  math  lesson.   The  interview  instrument  consisted  of 
twelve  items  related  to  planning,  teaching,  and  evaluating 
the  observed  teaching  segment. 

Following  the  interview,  raters  (doctoral  students  in 
Elementary/Early  Childhood  Education)  scored  responses  acord- 
ing  to  the  following  criteria: 

Reflective  Response:   (1  point)  A  response  characterized 
by  careful  consideration  given  to  the  item  in  question. 
A  reflective  response  might  include  such  indicators  as: 
appeal  to  one  or  more  theoretic  sources  (e.g.,  learn- 
ing theory,  developmental  theory),  appropriateness  for 
given  students,  societal  concerns,  support  from  intern- 
alized value  judgments  based  on  specific  criteria  (e.g., 
John  and  Mary  understood  because...). 

Non-Reflective  Response:   (0  points)   A  response 
indicates  little  or  no  consideration  was  given  for  the 
item  in  question.   Indicators  might  include:   conve- 
nience responses  such  as  "the  materials  were  easy  to 
gather,"  blind  faith  in  materials,  manuals  and/or 
cooperating  teachers,  and  global  justification  (e.g. , 
"they"  needed  it). 

Results 

Figure  1  shows  the  percentage  of  responses  which  were 
rated  as  having  provided  evidence  of  reflection.   The  mean 
combined  reflection  percent  was  21.   Looking  at  combined 
percentages  for  individual  subjects,  students  A  and  D  were 
clearly  rated  superior. 
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Percent  of  Responses  Rated  Reflective 
(Combined,  Reading,  Math) 


The  mean  reflection  percent  of  the  total  group  reading 
and  math  was  15  and  28  respectively.   Analysis  of  individual 
reading/math  ratings  presented  in  Figure  1  indicates  that 
subjects  A  and  D  were  rated  nearly  three  times  more  reflective 
in  math.   Subjects  B  and  C  obtained  the  same  score  for  both 
interviews.   E's  score  was  better  in  math. 
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Figure  2  represents  a  comparison  of  reflection  scores 
in  the  planning,  teaching  and  evaluative  dimensions  of 
reading  and  math. 


Planning 


Reading   [  | 


Math 


Teaching   Evaluating 
Dimensions 
Figure  2 


Reflection  percent  in  Planning,  Teaching,  and  Evaluative 
Dimensions  in   Reading  and  Math 

In  math,  students  were  most  reflective  about  planning 
concerns  and  least  reflective  about  interactive  teaching. 
In  reading,  students  were  most  reflective  about  teaching 
aspects  and  least  reflective  about  planning  concerns.  Chi- 
square  analysis  of  independence  indicated  a  significant 
difference  (p<.02)  between  reading  and  math  scores  in  the 
planning  dimension. 

Discussion 

Before  discussing  the  results,  it  must  first  of  all  be 
recognized  that  little  definitive  inference  can  be  postulated 
from  a  study  of  this  scope.   Among  the  most  notable  delimit- 
ations numbered:   (1)  the  sample  was  very  small,  (2)  the 
interview  instrument  was  not  tested  for  validity,  and 
(3)  scores  may  have  been  deflated  by  constraints  inherent 
in  the  student  teaching  process  as  described  by  Innaccone 
(1963),  Jacobs  (1968),  Katz  (1972),  Fuller  and  Brown  (1974), 
Cogan  (1974)  and  others. 

In  answer  to  the  research  questions,  results  indicated 
that  (1)  subjects  were  rated  as  having  responded  reflectively 
on  21%  of  the  items  (responses  of  subjects  A  and  D  greatly 
inflated  this  percentage)  (2)  three  of  the  five  subjects 
scored  higher  in  math  than  reading  while  two  subjects  ob- 
tained the  same  score  in  both  the  reading  and  math  inter- 
views, and  (3)  student  responses  to  math  interviews  were 
rated  significantly  (p<.02)  more  reflective  in  the  planning 
dimension  than  in  the  teaching  and  evaluative  areas.   No 
significant  difference  between  reading  and  math  scores  was 
found  in  the  teaching  and  evaluative  dimensions. 


-14- 


Following  analysis  of  results,  an  attempt  was  made  to 
look  at  a  few  variables  which  may  have  accounted  for  differ- 
ences in  scores.   First  of  all,  relationships  were  examined 
between  reflection  scores  and  traditional  indicators  of 
success  in  formal  academic  course  work.   Looking  specifically 
at  differences  between  reading  and  math  scores,  a  compari- 
son was  made  to  see  if  students  who  did  well  in  coursework 
(as  measured  by  grades)  were  rated  more  reflective  in  teach- 
ing the  respective  content  than  those  who  did  less  well. 
Table  1  shows  that  all  students  (except  E)  had  taken  formal 
academic  coursework  in  both  the  teaching  of  reading  (El.  Ed. 
336)  and  Math  for  Elementary  Teachers  (Math  202). 

Table  1 

Grades  Received  in  Teaching  of 
Reading  and  Math 


Student 

A 

B 

C 

D 

E 

Reading  Grade 

A 

B 

C 

B 

B 

Math  Grade 

A 

C 

D 

C 

- 

Reflection 

Rank 

1 

3 

4 

2 

5 

Rank-order  correlation  coefficients  computed  with  Table  1 
data  resulted  in  high  positive  correlations  between  reflec- 
tion rank/grade  rank  in  reading  (+. 70)  and  reflection  rank/ 
math  grade  rank  (+85).   With  an  n  of  five,  these  coefficient 
did  not  reach  the  critical  value  of  .9  needed  for  signifi- 
cance at  the  .10  level.   However,  since  coefficients 
approached  the  critical  value,  further  research  is  needed 
with  larger  sample  sizes. 

Reflection  score  rank  of  each  student  was  also  paired 
with  ratings  in  El.  Ed.  232  (Supervised  Student  Teaching), 
El.  Ed.  230  (Student  Teaching  Problems  and  Issues  Seminar) 
and  GPA.   No  consistent  relationship  between  these  variables 
was  found. 

In  addition  to  examination  of  formal  academic  indicators, 
reflection  ranks  were  compared  with  teaching  variables  such 
as:   teacher-student  ratios  during  the  observed  lessons, 
questioning  styles,  and  ratings  of  cooperating  teachers. 
Analysis  of  the  teacher-student  ratio  revealed  that  each 
reading  group  had  a  ratio  of  1:7  or  less.   In  math,  however, 
the  ratio  increased  to  1:21  or  greater  for  subjects  who 
ranked  3rd,  4th,  and  5th  in  reflection.   Subjects  A  and  D 
who  were  rated  most  reflective  retained  the  1:7  or  lower 
ratio  in  the  math  lesson. 
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Questions  asked  by  subjects  during  the  observed  lesson 
were  rated  as  (1)  those  which  elicited  a  single  response  and 
(2)  those  which  elicited  a  range  of  learner  responses.   Data 
showed  that  subjects  asked  more  range-response  questions  in 
math.   Also,  although  ranked  range-response  questions  in 
reading  were  not  consistently  related  to  reflection  rank, 
reflection  rank  and  range-response  rank  in  math  corresponded. 

Finally,  many  researchers  such  as  Perrodin  (1961), 
Innaccone  (1963),  Yee  (1969),  Lowther  (1970),  Fuller  and 
Brown  (1975)  and  others  have  suggested  that  cooperating 
teachers  may  significantly  effect  attitudes  and  behaviors 
of  novice  pedagogues.   To  obtain  an  indication  of  this  effect 
in  the  present  study,  the  investigator  rated  each  subject's 
cooperating  teacher  on  three  characteristics — modeling 
behavior,  feedback  provided  to  student  teachers,  and  the 
relative  amount  of  freedom  student  teachers  were  given  to 
implement  their  ideas.   Each  indicator  was  then  rated  with 
the  scale:   3  (Good),  2  (Fair),  and  1  (Low).   Table  2  pairs 
student  teacher  reflection  rank  with  corresponding  mean 
cooperating  teacher  rank  on  the  above  indicators. 

Table  2 

Student  Teacher  Reflection  Rank  and 
Mean  Cooperating  Teacher  Rank 


Student  Teacher 

A 

B 

C 

D 

E 

Student  Teacher 

Reflection  Rank 

1 

3 

4 

2 

5 

Mean  Cooperating 

Teacher  Rank 

1 

3.5 

2 

3.5 

5 

Although  ranks  were  not  perfectly  correlated,  student  teachers 
with  the  highest  and  lowest  reflection  ranks  had  cooperating 
teachers  who  corresponded  to  those  same  ranks. 

Summary 

The  present  study  was  an  attempt  to  obtain  an  indica- 
tion of  reflection  exhibited  by  student  teachers  during  the 
teaching  process.   Results  indicated  that:   (1)  subjects 
were  rated  as  having  responded  reflectively  on  21%  of  the 
total  items,  (2) students  tended  to  be  rated  more  reflective 
in  math  than  reading,  and  (3)  responses  to  math  interviews 
were  rated  significantly  (p<.02)  more  reflective  than  read- 
ing responses  in  the  "planning"  dimension  of  teaching. 
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Finally,  if  a  major  goal  of  teacher  education  institu- 
tions is  to  prepare  teachers  for  making  responsible  educa- 
tional decisions,  the  study  suggested  concern  that  program 
elements  nurture  reflective  dispositions  in  students. 
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TEST  RESULTS   REVISITED 

Patricia  T.  Scheyer 
Early  Childhood  Education 

In  many  school  systems,  the  use  of  standardized  tests  has 
long  been  the  major  evaluation  strategy.   Such  assessment  of 
student  achievement,  aptitude,  or  intelligence  is  usually  esta- 
blished and  regulated  by  state  and  local  requirements.   Though 
classroom  teachers  often  administer  the  tests,  they  seldom  re- 
ceive results  more  informative  than  single  scores  for  students. 
While  some  critics  call  for  reform  or  abandonment  of  testing 
practices,  the  teachers  interviewed  in  this  study  have  begun 
developing  alternative  ways  of  evaluating  children  in  the  con- 
text of  the  classroom  environment. 

Objections  to  testing  have  been  voiced  in  unequivocal  terms 
by  many  educators.   Karier  (1973)  scored  intelligence  and 
achievement  tests  as  instruments  of  control  in  the  corporate 
state,  maintaining  within  the  schools  a  system  of  sorting  and 
classifying  people  into  a  hierarchy  of  social  and  economic 
classes.   Zacharias  (1975)  claimed  tests  have  an  imprisoning 
and  destructive  effect  on  children  when  a  child's  worth  is  re- 
duced to  a  single  test  score  placed  on  a  single  line  order  of 
intelligence  or  achievement,  particularly  since  there  is  no 
agreement  about  what  test  items  measure. 

Tests  are  criticized  for  middle-class  bias,  for  rewarding 
speed  and  conventional  thinking,  for  labeling  children,  and 
for  measuring  children's  cultural  and  experiential  backgrounds. 
Although  there  may  be  little  agreement  about  what  test  items 
measure,  teaching  test  items  is  one  practice  associated  with 
raising  students'  scores.   House  (1973)  found  testing  in  the 
context  of  evaluation  and  accountability  as  simplistic,  un- 
workable, contrary  to  empirical  findings  and  ultimately  im- 
moral. 

Test  scores  may  be  valid  for  discriminating  among  indivi- 
dual students  within  a  classroom  group  (Stake,  1968).   Their 
validity  is  low,  inapplicable  or  unestablished  for  other  com- 
mon uses  such  as  diagnostic  indicators,  comparing  schools,  or 
evaluating  the  competency  or  effectiveness  of  teachers  or  the 
suitability  of  curricula  for  an  entire  group  (Stake,  1975). 
Tests  tell  the  teacher  little  about  what  is  happening  in  the 
program  as  students  are  learning.   Hastings,  Runkel,  et  al., 
(1960)  found  that  in  situations  where  teachers  had  available 
both  test  results  and  teacher  assessments  of  students,  they 
often  vetoed  the  test  results  and  preferred  teachers'  judg- 
ments instead.   Hotvedt  (1974)  concluded  that  most  teachers 
actually  do  not  use  available  results. 
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Hoffman  (1975)  called  for  any  kind  of  testing  or  evaluation 
which  allows  the  student  to  do  something  himself,  and  where  the 
primary  concern  is  with  the  reasoning  process  rather  than  the 
answer.   Both  theoretical  and  applied  alternatives  to  testing 
student  outcome  have  been  suggested  in  the  literature,  notably 
by  Eisner  (1969,  1974),  Gallagher  (1970a,  1970b),  Stake  (1967, 
1975),  Bussis,  Chittendon  and  Amarel  (1973,  1975,  1976),  Perrone 
(1972,  1975),  Carini  (1975),  Duckworth  (1973),  Wolf  (1971), 
Scheyer  and  Stake  (1976),  Kemmis  (1974),  Elliott  (1975,  1976), 
Glass  (1974),  Barclay  (1974),  and  Anderson  and  Walberg  (1974). 

Program  evaluators '  concern  has  grown  recently  for  under- 
standing more  about  teaching  and  learning  in  the  context  of 
processes  and  interactions  within  the  classroom  environment 
among  students,  teachers,  curriculum  and  materials.   This  po- 
tentially rich  source  of  contextual  data  was  explored  in  this 
study  by  asking  teachers  how  they  feel  about  the  policy  and 
practice  of  testing  and  the  use  of  test  results.   The  teachers' 
comments  support  many  of  the  claims  and  criticisms  in  the  liter- 
ature . 

The  challenges  to  testing  of  Hotvedt,  Hoffman  and  others 
were  explored  in  Ravenwood,  Ohio,  a  suburban  school  district 
where  policy  requires  standardized  IQ  and  achievement  testing 
in  grades  three  and  six.   They  also  require  other  kinds  of  tests 
from  time  to  time,  with  an  increasing  number  administered  by 
the  central  pupil  testing  department.   Beyond  that,  local 
option  of  the  principal  has  left  to  teachers  considerable  dis- 
cretion in  testing  practice.   The  district  test  scores  of  Raven- 
wood  are  well  above  the  national  average.   Its  middle  to  upper 
class  children  are  generally  regarded  as  winners,  the  kind  of 
children  the  schools  and  testing  system  are  designed  to  bene- 
fit. 

All  the  quotations  in  this  study  are  reported  from  inter- 
views with  three  early  childhood  teachers  conducted  at  Wood- 
shill  Elementary  School  in  November  1975.   Mrs.  Bernstein 
teaches  two  half -day  sessions  of  kindergarten,  Mrs.  King  first 
grade  and  Mrs.  Langley  as  combination  class  of  first  and  second 
graders.  All  three  teachers  are  experienced,  dedicated  profes- 
sionals.  Their  classroom  can  be  characterized  as  informal,  se- 
mi-structured, and  pluralistic--rich  in  opportunities  to  learn 
with  students  participating  in  a  variety  of  activities  simulta- 
neously. 

The  interviews  were  opened  with,  "I'm  interested  in  the 
uses  teachers  make  of  tests  and  test  results.   Testing  means 
anything  you  define  as  testing  or  a  way  of  getting  information 
about  children,  from  standardized  tests  to  any  kind  of  teacher 
assessment.   Do  you  use  tests?" 
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The  ensuing  interviews  were  taped  for  later  analysis.   The 
discussions  focused  primarily  on  the  seven  issues  which  emerged 
during  the  interviews:  evaluation  policy  in  an  elite  community; 
cooperative  policy  making  for  influencing  change;  harmful  effects 
of  testing;  fairness;  individuality  and  standardized  tests; 
teacher  assessment  vs.  standardized  testing;  and  evaluation  in- 
tegrated with  curriculum.   Further  analysis  has   synthesized 
these  recurring  themes  into  the  three  predominant  issues: 
evaluation  policy;  harmful  effects  of  testing;  and  individual- 
ized evaluation. 

Evaluation  policy:   "Admission  to  Harvard  and  super  gain  scores" 

As  schools  tend  to  reflect  the  communities  they  serve, 
testing  policies  at  Woodshill  School  appear  to  reflect  the 
values  of  the  community's  parents.   The  teachers  think  many 
parents  attained  Ravenwood  status  by  improving  on  the  economic 
and  social  status  of  their  own  parents.   This  status  may  in 
part  be  due  to  more  education.   Testing  has  such  an  impact  on 
schooling  that  the  teachers  say  the  parents  tested  themselves 
upward  into  Ravenwood. 

The  reachers  regard  many  Ravenwood  parents  as  elitists  with 
the  highest  expectations  for  their  children,  citing  such  un- 
realistic goals  as  admission  to  Harvard  or  "super  gain  scores." 
Parents'  goals  have  created  a  climate  in  which  demands  for 
accountability  exert  pressures  on  the  school  administration. 
"We  have  a  200%  PTA,"  noted  Mrs.  Bernstein.   "It  is  possible  we 
could  have  500%.   They  are  overly,  super  involved." 

Demands  for  accountability  and  recently  purchased  program- 
med curriculum  materials  with  tests  embedded  in  them  both  contri- 
bute to  escalating  the  amount  of  testing.   This  creates  pressure 
on  teachers  and  children.   Mrs.  King  fears  that  accountability 
pressures  and  resulting  tests  or  paper  work  may  increase  an- 
xiety, teaching  children  to  hate  school.   She  feels  the  situa- 
tion is  compounded  by  the  administration  policies  that  "try 
to  meet  children's  needs  by  being  accountable  to  parents."  Mrs. 
Bernstein  is  concerned  that  parents  "worry  and  get  uptight," 
and  may  misinterpret  scores  to  mean  their  child  is  less  bright 
than  they  would  like,  or  that  he  is  enrolled  in  the  wrong  school, 
with  less  competent  peers.   These  feelings  ultimately  are  trans- 
ferred to  the  child. 

The  teachers  would  like  to  communicate  more  closely  with 
administrators  and  parents  on  evaluation  policy,  the  realities 
of  its  practice,  and  its  effect  on  students.   "The  trouble  is, 
the  district  makes  these  policy  changes  without  asking  the 
teachers,  and  it's  hard  to  work  together,  even  with  the 
specialists ." 
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Mrs.  Bernstein  holds  that  the  "concept  of  primary  educa- 
tion has  changed.   In  the  1960 's  policies  were  freer.   Teachers 
had  more  responsibility  for  developing  curriculum  and  there 
were  more  real  life  experiences,  more  activities  and  trips  as 
the  basis  for  learning.   There  was  little  concern  about  academic 
skills  and  assessment  for  first  grade  readiness."  Now,  the 
teachers  are  not  consulted  with  regard  to  the  district's  selec- 
tion of  required  curriculum  materials  and  kits  or  the  tests 
associated  with  them  although  they  have  the  responsibility  to 
implement  these  curricula  and  administer  the  tests. 

Teachers  and  parents  may  share  similar  hopes  for  children's 
development,  but  differ  in  their  approaches.  An  example  of 
working  more  closely  with  parents  is  Mrs.  Langley's  approach  of 
enlisting  parents  as  classroom  volunteers  to  help  them  under- 
stand more  about  child  development,  learning  and  within  reason 
what  children  can  or  cannot  do. 

The  teachers  resist  evaluation  policy  that  is  based  on 
standardized  testing  because  they  feel  it  is  unfair  and  mini- 
mized individuality.   They  would  like  to  avoid  using  tests, 
claiming  that  harmful  effects  on  children  outweigh  the  advantages 
to  parents  or  teachers. 

Harmful  effects:   Teaching  for  creativity  and  divergent  thinking, 
measuring  convergence 

The  testing  situation  is  viewed  by  these  teachers  as 
"socially  and  emotionally  damaging  for  children  to  experience." 
Mrs.  Langley  said,  "There  are  so  many  variables  in  a  test 
situation  you  can't  give  the  same  test  to  everyone.   One  may 
not  have  gotten  started.   Movement  distracts  some.   Maybe  a 
child  didn't  have  breakfast.   Maybe  the  test  is  going  to  tell 
him  how  he  is  incompetent  or  destroy  his  self-image  and  he 
knows  that.   You  know  tests  test  for  the  average,  and  the  way 
they  are  scored,  50%  have  to  be  below  the  norm."  Mrs.  King 
describes  a  problem  with  new  kindergarten  tests  that  are  used 
to  predict  performance  and  determine  pupils'  placement  in  spe- 
cific programs  in  first  grade.   "The  tests ...  include  such  diffi- 
cult skills  that  only  a  very  unusual  child  could  get  very  far 
into  them.   That  is  hard  on  their  self-esteem  as  they  take  the 
test." 

Critical  of  tests  themselves,  the  teachers  object  to  the 
narrow  range  and  selection  of  skills,  variables  and  "not  quite 
right  answers"  that  are  acceptable  on  test  items.   The  teachers 
see  the  children  in  more  complexity  than  tests  can  measure. 
For  this  reason,  they  are  skeptical  of  tests'  accuracy  and  use- 
fulness.  Children  are  developing  and  changing  all  the  time, 
even  if  idiosyncratically.   "We  do  not  know  anything  for  sure, 
even  with  pre- test,  post-test  procedures,"   said  Mrs.  King. 
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"We  teach  for  creativity  and  divergent  thinking  when  tests  mea- 
sure convergence.  Tests  freeze  children  in  a  score  even  though 
they  go  right  on  developing." 

Labeling  children  with  results  of  questionable  test  items 
is  basically  unfair  and  too  controlling.   The  labels  placed  on 
them  in  the  early  grades  bias  subsequent  teachers'  expectations, 
the  peer  relationships,  the  parents,  and  the  children's  self- 
expectations.   "The  self-fulfilling  prophesy  works,"  Mrs. 
Langley  is  convinced.  Eventually,  test  scores  and  labels  track 
children  educationally  into  classes  from  which  it  becomes  in- 
creasingly difficult  to  extricate  themselves.   Early  childhood 
teachers,  they  asserted,  should  be  reluctant  to  begin  this  pro- 
cess of  stigmatizing  children. 

"We  want  to  give  children  every  opportunity  to  learn  and 
grow  without  putting  labels  on  them.   I  try  to  give  all  those 
opportunities  before  I'm  willing  to  call  others  to  come  in  here 
and  take  the  child  apart,"  said  Mrs.  Bernstein.   "I  believe  so 
much  learning  has  to  do  with  tnaturational  experiences  and  grow- 
ing.  Yet,  we  are  tracking  children  early  with  standardized 
tests ." 

The  teachers  object  to  sharing  their  power  with  testing 
specialists,  they  think  that  they  abuse  that  power.   Teachers 
feel  it  is  an  invasion  of  their  territory,  devastating  a  per- 
centage of  the  children  by  lowering  their  self-esteem  and 
leaving  the  teacher  to  pick  up  the  pieces.   "The  real  question," 
added  Mrs.  King,  "is  how  and  who  uses  the  data  from  tests." 
Not  only  are  teachers  unable  to  obtain  test  results  they  wanted 
to  use  for  diagnosis  and  remediation  follow-up,  but  specialists 
with  access  to  the  data  have  also  failed  to  make  it  work  for 
children.   Mrs.  King  holds  that  "at  the  end  of  the  year  testing 
specialists  sent  lists  of  children  who  did  poorly  on  this  test. 
They  announced  they  would  work  individually  with  those  children 
in  perception.   We  never  saw  them."  All  teachers  reported 
getting  results  back  so  late  that  students  were  already  doing 
the  skills  the  tests  showed  they  could  not  do,  especially  in 
the  area  of  perceptual  skills. 

"Testing  doesn't  help  the  child.   It  helps  the  teacher, 
but  we  get  the  results  back  so  late,"  said  Mrs.  Bernstein.  "The 
follow-up  is  really  the  teacher's  responsibility."  Results  are 
used  for  political  and  social  purposes  in  the  schools  and  the 
community,  but  diagnostic  uses  are  thwarted.   Mrs.  Langley 
argues  testing  "is  not  useful,  is  costly  and  very  time  consum- 
ing." 

To  alleviate  the  futility  of  gathering  results  later  mis- 
used, they  advocate  an  open  communication  arrangement  with  the 
testing  office  to  establish  that  particular  children's  problems 
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have  diminished.   "That's  one  way  to  reduce  the  stigma,"  said 
Mrs.  Langley.   In  general,  the  teachers  say  they  have  seen 
little  evidence  of  constructive  use  of  standardized  testing  to 
outweigh  the  harmful  effects. 

Individualized  evaluation:   "Did  it  work  with  this  child?" 

Alternative  ways  of  assessing  children  in  the  context  of 
the  learning  situation  are  being  developed  by  the  teacher  as  an 
integral  part  of  their  teaching.   School  can  make  a  difference, 
they  believe,  if  children  are  viewed  as  individuals.  The  teachers 
see  evaluation  as  an  ongoing,  formative  part  of  the  classroom 
process.   They  emphasize  the  relationship  between  the  quality 
of  content  in  the  learning  activities  and  the  quality  of  what 
individual  children  do  to  learn  and  produce.   The  basis  of 
early  childhood  curriculum  for  these  teachers  seems  to  be  the 
teaching  of  powerful  concepts.   To  promote  the  development  of 
a  real  understanding  they  offer  children  a  variety  of  learning 
experiences  for  each  concept.   Their  most  frequent  questions  to 
themselves  are,  "Did  it  work  with  this  child?  What  else  could 
I  do  to  make  it  more  effective?   Can  he  also  do  this  alternative 
task?"  The  teachers  say  they  try  to  cross-check  a  child's  abi- 
lity to  grasp  the  concept  by  his  performance  in  several  alter- 
native tasks. 

What  the  teachers  think  is  worth  evaluating  and  their 
methodology  are  tied  together.   In  attempting  to  individualize 
both  instruction  and  evaluation,  frequent  observations  and 
interviews  with  children  guide  the  teacher's  day  to  day  plan- 
ning. Assessing  what  and  how  well  children  are  doing,  at  what 
skills  they  are  successful  or  weak,  indicates  to  the  teacher 
what  is  appropriate  for  the  child  to  try  next.   "There  is  so 
much  variation  in  children  and  the  way  they  learn.  A  teacher 
can  accommodate  a  lot  of  different  right  answers  from  different 
children  for  any  given  concept,  which  a  test  can't  do.   You  can 
also  retest  in  a  sense  when  you  help  them  move  to  the  next  step," 
said  Mrs.  Langley. 

Mrs.  Langley  described  her  method  of  watching  the  child's 
learning  style,  how  self -directed  the  child  is;  how  he  makes 
choices,  plans  and  decisions  about  "what  he  thinks  the  next 
step  is.   Maybe  that's  motivation.   You  can't  test  it,  but  it's 
here  to  observe,"  she  said. 

Mrs.  King  recalled,  "I  watch  and  observe  and  jot  down  notes 
on  them--anecdotes ,  thoughts,  comments --once  a  week  or  so, 
sometimes  once  a  day.   I  look  at  social  and  emotional  develop- 
ment, attention  span,  maturity,  physical  development,  percep- 
tion, academic  progress,  ability  to  make  decisions,  to  analyze, 
discuss  and  to  ask  questions.  We  do  a  lot  of  sharing  and  dis- 
cussion.  It's  interesting  how  accurately  the  kinds  and  variety 
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of  questions,  answers  and  dialogue  I'm  getting  are  indicators 
of  where  the  child  is.   Giving  direct  academic  help  or  social/ 
emotional  help  is  an  enigma,  because  you  can't  really  tell  the 
direct  results.   The  answers  seem  to  lie  in  providing  a  huge 
variety  of  experiences,  rather  than  in  pegging  the  children." 

The  teachers  prefer  information  gathered  in  the  context  of 
a  more  who lis tic  evaluation  of  the  child's  learning  by  someone 
who  knows  him  and  works  closely.   They  try  to  consider  their 
teaching  and  the  environmental  as  well.   They  have  developed 
their  own  kind  of  tests  using  categories  of  subject  areas  and 
abilities  they  find  relevant  to  learning  and  teaching.   They  ob- 
serve, interview  and  make  judgments  about  achievements  in  parti- 
cular situations.   Using  anecdotal  records,  they  gather  compara- 
tive data  on  each  child  on  individual  skills .   Often  they  focus 
on  such  evaluative  criteria  as  level  of  participation,  level  of 
conceptual  complexity,  ability  to  verbalize  (spoken  or  written) 
or  make  drawings  about  learning  experiences ,  and  improvements 
in  language  and  discourse. 

Occasionally  they  use  a  checklist  of  various  skills  devel- 
oped partly  from  the  testing.   Mrs.  Bernstein  acknowledged  "that 
the  perceptual  testing  improved  my  teaching,  made  me  more  aware 
and  observant  about  perceptual  elements  in  the  child."  The 
teachers  gave  examples  of  how  they  evaluate  children's  work,  and 
illustrated  making  inferences  about  perception,  motor  sensory 
development,  reading,  math  and  interpersonal  relationships. 

For  reading  assessment,  Mrs.  Langley  described  her  proce- 
dure.  "I'd  take  the  child  to  the  library  to  pick  out  books  he 
was  interested  in  and  go  from  there.   I'd  confer  with  him  as  he 
read,  and  work  until  I  found  any  special  problems."  The  rela- 
tionship between  language  development  and  reading  is  regarded 
as  so  critical  by  these  teachers  that  they  stress  experiential 
reading.   The  evaluation  most  pertinent  is  program  and  child 
specific.   They  evaluate  vocabulary,  syntax,  and  comprehension, 
using  phonetic  or  sight  approaches  as  appropriate  to  the  indivi- 
dual.  The  teaching  corrolary  is  the  emphasis  on  reading  and 
using  many  books,  and  offering  chances  for  discourse  in  the 
classroom.   Mrs.  Bernstein  said,  "Field  trips  give  you  dialogue, 
questions,  direct  experiences,  and  then  all  the  real  thinking. 
The  sense  of  authorship  from  recalling  and  writing  stories 
brings  kvell.   That  means  to  be  so  pleased  inside  that  you're 
smiling  with  your  own  accomplishments." 

The  math  evaluation  also  consists  of  working  individually 
with  objects  and  activities  in  which  math  experiences  are  em- 
bedded.  Their  teaching  stresses  manipulative  experiences  prior 
to  paper  and  pencil.   Symbolization,  geometric  concepts,  sets 
and  operations  are  all  taught  and  assessed  with  the  same  mater- 
ials.  Mrs.  King  believes  understanding  the  concepts  behind 
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numbers  far  more  natural  for  kindergarten  aged  children  then 
learning  sounds  and  phonics.   "If  a  child  has  a  pretty  good  idea 
about  numbers  including  the  operations,  I  can  be  pretty  sure  he 
or  she  will  probably  function  quite  well  in  other  areas  too, 
especially  reading." 

Children's  paintings  and  drawings  are  analyzed  regularly 
for  content.   "They  represent  how  the  child  feels  about  the 
world  and  himself.   I  get  a  sense  of  emotional  maturity,  of 
language  development,  of  motivation,  and  of  motor-sensory  and 
perceptual  development,"  noted  Mrs.  Langley,  "but  especially  of 
aesthetic  awareness." 

Motor  perceptual  abilities  are  assessed  by  direct  observa- 
tion of  physical  abilities  in  hopping,  skipping  and  jumping. 
Such  aptitudes  as  spatial  relationships  are  judged  by  behaviors 
like  bumping  into  objects,  clumsiness,  or  how  a  child  sits  in 
the  group.   Some  children  are  disoriented  in  the  class  or 
school.   Mrs.  Bernstein  gives  practice  to  such  children  by 
sending  them  on  missions  to  other  places  in  school. 

The  teachers  argue  that  it  is  important  to  evaluate 
children's  interpersonal  relationships  —  the  ability  to  trust, 
respect  each  other  and  communicate.   Those  with  interpersonal 
problems  often  develop  learning  problems  later.   For  certain 
children  with  problems,  the  advantages  of  a  wholistic  evalua- 
tion approach  are  particularly  evident,  and  point  up  the  in- 
adequacies of  cognitive,  test-based-assessments.   Such  children 
are  sometimes  strong  academically,  but  need  to  be  referred  for 
special  psychological  services. 

The  results  of  this  ongoing,  informal  teacher  evaluation 
are  reported  to  parents  in  ways  that  emphasize  children's  in- 
dividuality.  They  keep  a  portfolio  of  anecdotal  notes  and  ob- 
servations, examples  of  the  child's  written  work,  and  before 
and  after  drawings  from  beginning  to  end  of  the  year.   They 
also  present  a  "progress  report"  at  the  conference  about  per- 
formance, abilities  and  idiosyncracies .   "It  is  a  summary  of 
what  I  know  of  the  child  in  a  written  case  study,"  reported 
Mrs.  Langley.   Teachers  prefer  analytic  description  to  norma- 
tive report  cards,  they  said.   Referring  to  the  "myth  of  test 
norms"  they  regard  report  cards  as  competitive  and  unfair  to 
the  children. 

The  teachers  are  alert  to  how  they  use  elements  in  the 
curriculum  as  indicators  for  evaluation,  but  they  see  this  as 
very  different  than  teaching  to  tests.  Their  indicators  are 
varied  and  situational,  the  assessment  is  conducted  in  a  nat- 
uralistic setting,  and  it  is  supportive  rather  than  threaten- 
ing. "It  is  no  different  than  what  we  do  every  day,  so  that 
all  children  have  the  equal  chance  to  do  their  best." 
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The  teachers  trust  their  own  assessment  of  children  in  the 
context  of  the  classroom  as  valid,  more  so  than  tests.   In  con- 
trast to  the  teachers'  individualized  evaluation,  standardized 
tests  seem  to  them  to  deny  certain  individual  characteristics 
and  circumstances.   "It  seems  to  me  we've  seen  over  and  over 
again  that  teachers  can  evaluate  children  as  well  or  better  than 
tests,  and  since  they  do  it  in  a  supportive  way  it  is  not  so 
damaging."  The  judgments  they  make  are  related  to  their  teach- 
ing, and  that  "makes  it  more  fair  to  the  child  because  it  im- 
proves the  quality  of  what  happens  for  each  child  in  the  class- 
room," claimed  Mrs.  King. 

Parents  in  Ravenwood  expect  their  children  are  college 
bound.   If  they  want  teaching  for  creativity  and  divergent 
thinking,  if  they  want  direct  field  experiences  that  develop 
dialogue  and  question-asking,  that  produce  the  thinking  children 
can  write  and  read  about,  it  takes  a  certain  style  of  teaching. 
That  style  requires  a  harmonious  and  appropriate  style  of  eval- 
uation that  is  embedded  in  the  teaching,  the  teachers  asserted. 

They  see  individuality  and  growth  partly  as  a  function  of 
a  child's  response  to  many  diverse  learning  experiences  in- 
tended to  extend  knowledge,  creativity,  and  aesthetic  sensiti- 
vity and  critical  thinking.   Because  they  cannot  predict  all 
the  learning  that  will  take  place,  they  try  to  provide  diverse 
activities,  "If  we  were  to  narrow  teaching  to  basic  skills  that 
are  test  items,  we  would  never  know  what  talent  we  missed,  what 
characteristics  could  have  developed,  what  appreciations,  what 
relationships,"  concluded  Mrs.  King. 
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CONVERSATIONS  WITH  MYSELF 

OR 

WHAT'S  A  MICE  GIRL  LIKE  YOU  VOING  IN  EVALUATION? 

Carol  Hodges  Wardrop 
Elementary  Education 

?n.e,jace. 

Once  upon  a  time  a  long  time  ago   [Augsut  J 97 3)  a  "natheA 
trusting,   naive.,   unsophisticated,  gullible  and  unwonldly" 
glnZ  entexed  the  linn1  &  dm   [otherwise  known  as  the  xeal 
would  oh  evaluation!) .     She  was  neveA  seen  on.  heand  ^xom  again. 
Some,  &olks  say  they've,  seen  a  "woxldly,  sophisticated,   schem- 
ing shxewd  and  expexlenced"  lady  who  looks  vaguely  ^amUlon 
neon.  the.  entrance  to  that  same.  den.     But  that  couldn't  be. 
hex...  .could  it? 

Hmm...J  think  I  see.  hex  now,   I  wonden.. 

"Excuse,  me,,    'woxldly,  sophisticated,   scheming  shxewd  and 
expexlenced'  lady.     Vou  would' nt  happen  to  be.  that  'xathex 
trusting,   naive,,   unsophisticated,  gullible,  and  unworldly' 
glnl  who  entexed  the,  dm  two  yeaxs  ago,  would  you? 

you  would! !     What  was  a  nice,  gixl  tike,  you  doing  In 
evaluation???" 

"Sometimes  I  wonder  about  that  myself.   The  naive  girl 
that  I  once  was  certainly  did  not  expect  evaluation  to  be 
such  a  socio-political-economic-administrative-legal-moral 
conglomeration  that  I  now  know  it  to  be.   Have  you  ever 
heard  of  an  evaluator  named  Dan  Stufflebeam?  He  wrote  a 
paper  on  meta-evaluation  in  which  he  discusses  six  classes 
of  problems  that  he  believes  may  jeopardize  an  evaluation 
study  if  they  are  not  thought  through  before  the  study 
begins.   These  classes  are  conceptual,  socio-political, 
contractual/legal,  technical,  administrative  and  moral/ 
ethical.   If  you  really  want  to  know  what  happens  to  a  nice 
girl  in  evaluation,  I'll  have  to  discuss  a  few  of  those 
problems. 

In  order  to  understand  my  story,  I'll  have  to  take  you 
back  to  1972  when  the  American  Education  Foundation  (AEF)* 
awarded  contracts  to  two  ambitious  efforts  to  implement 
large-scale  and  long-term  demonstrations  of  their  capabil- 
ities as  systems  of  computer-assisted  instruction.   One  of 


*Although  this  story  is  true,  all  names  have  been 
changed. 
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them,  Information  Computer  Television  (ICT),  was  being  carried 
out  by  the  RET  Corporation  in  Southcity.   The  other,  known  as 
Logical  Information  Programming  (LIP),  was  being  developed  by 
the  Computer  Laboratory  of  Big  University.   The  Testing 
Service  of  America  (TSA) ,  located  in  Easternton,  was  awarded 
a  contract  by  AEF  at  the  same  time  to  conduct  an  extensive 
and  through  evaluation  of  each  of  the  two  systems. 

The  evaluation  of  the  ICT  and  LIP  programs  was  designed 
to  focus  on  three  major  components  of  the  demonstration  pro- 
jects— their  costs,  performance  and  educational  effectiveness. 
The  classroom  demonstrations  and  the  evaluation  of  them  were 
expected  to  begin  in  September  1973  and  continue  for  approx- 
imately two-and-a-half  years.   The  contract  called  upon  TSA 
to  send  quarterly,  annual  and  specified  special  reports  to 
AEF  on  a  regular  schedule.   In  addition,  RET  Corporation  and 
Big  University  were  to  receive  copies  of  the  reports.   By  May 
of  1973,  TSA  had  drafted  a  document  describing  a  plan  for  the 
evaluation.   This  document  presented,  among  other  things,  the 
role  that  TSA  was  to  play  in  the  evaluation,  goals  and  be- 
havioral objectives,  sampling  plans,  instrumentation,  data 
analysis,  a  timetable  and  dissemination  of  the  results. 
Conceptually,  I  believe  it  was  very  well  thought  out." 

"Okay,   okay.     That' 6  all  uoeLt  and  good.     But  wheAQ,  (Viz 
you  -en  thZi,  p-aituAz  't/ua>tlng,   naive,   unAopkiAtXcated, 
gullible,  and  unworldly'  ginl?     What  mo6  happening  to  you? 
What  pant  did  you  play?    What  wa6  happening  out  In  the.  leal 
would  o^  evaluation? 

"Oh  yes,  I  did  promise  to  tell  you  what  happened,  didn't 
I?  It's  just  that  it's  so  much  easier  to  talk  about  plans 
than  it  is  to  make  them  work.   Extraneous  things  like  teachers, 
students  and  machines  that  don't  work  keep  getting  in  the  way. 
I  was  only  involved  with  one  component  of  the  evaluation,  so 
my  conversation  with  you  will  have  to  focus  on  the  evaluation 
of  the  elementary  school  math  and  reading  portion  of  the  LIP 
Evaluation.   I  really  know  very  little  about  the  other  parts 
of  the  LIP  evaluation  or  anything  about  the  ICT  portion. 

Because  TSA  is  located  in  Easternton,  the  evaluators 
there  decided  to  hire  someone  here  in  Middletown  to  act  as  a 
local  evaluation  coordinator.   That  became  my  job.   My  ini- 
tial task  was  to  talk  with  the  LIP  staff  who  were  working 
on  the  elementary  programs  in  order  to  ascertain  the  state 
of  the  curriculum.   I  was  also  to  discuss  with  them  their 
plans  for  involving  local  schools  and  teachers  in  the  field 
trials,  which  were  expected  to  begin  in  a  month.   I  also 
became  acquainted  with  the  computer  language  and  the  LIP 
computer  facilities.   During  my  first  week  on  the  job,  it 
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became  apparent   that  anyone  associated  with   the  evaluation 
and   "THAT  PLACE"    (TSA)   were   suspect.      I  had  no  place  to 
work,    no  desk,   no   equipment   and  no   supplies." 

"Oh,   oh!     You  Kan  mack  into  owl  oi  the  administ/uvtiv e 
pKoblem*  Stu&&lebeam  talk*  about  didn't  you?     TSA  had  made, 
no  pKtoK  pKovision  ^ok.  o^ice  &pace,   equipment  on.  materials? 
What  was  available.?     knswets  to  question*  tike,  that  can 
Keally  efi&ect  the.  ea*e  with  which  evaluation*  can  be. 
cahJited  out.     1-6  that  when  you  AtaKted  to  change,  -into  that 
'woKldly,  sophisticated,  6ch.emi.ng,   i>hn.ewd  and  experienced' 
lady?" 

"No,    not  yet.      I  was   still  rather  naive  and  believed   in 
things  like  the  TSA  'Fairy  Godmother.'      Anyway,    I  did  finally 
get  a  desk,    even  though  I  had   to  move  three  times   that    first 
year.       (A  year   later  when  another   local   evaluator  joined  me, 
she  only  had  to  move  twice!)      Unfortunately  we  never  did  get 
a  typewriter   or   secretarial  help.      Much  of   our   time   is 
wasted  Xeroxing   reports  which  we   send   to  TSA  to  be  typed.      I 
have  become  rather   fond  of   the  Xerox  machine  though. . .when 
you  are  labeled  as  an  evaluator,    you  sometimes  have   to   find 
your  friends   in  strange  places! ! !      You  are  right   though. 
Administrative  problems   such  as   location  and  facilities 
should  be  discussed  and  taken  care  of  before  the   evaluation 
begins. 

Anyway  that  was  just  a   small  problem.      After  working 
for  a  week  or  so   at   the  Computer  Laboratory,    it  became 
obvious   to  me   that    the   field   demonstrations  would  not   begin 
as   scheduled.      The  LIP  personnel  had  not   even  recruited 
teachers  yet.      Plans   for  the   implementation  had   changed 
frequently  since  February   1972  and   courseware  developments 
had   slipped   seriously.      These  uncertainties  complicated   the 
evaluation  plans.      Without  more  complete  information  on  the 
implementation,    including    the   distribution  and   placement   of 
terminals,    content   outlines  of  all   instructional  materials, 
and   identification  of   teachers  and  pupils  who  would  use   the 
system,    TSA' s   efforts  were  severely  hampered." 

"1  gueAi>  you  ate.  Kight.     How  can  you  evaluate,  a 
pKogKam  i&  there  isn't  anything  there  to  evaluate..     What 
did  the.  contract  bay  about  pK.ote.cti.ng  the.  evaluation  i{ 
there  were  delays  In  the.  product?    VoeAn't  Stu^lebeam  *ay 
all.  parties  involved  in  an  evaluation  Kequire  pK.ote.cti.on 
and  oAbunance?     What  happened?     Von' t  6top  now,   it*  just 
getting  interesting." 
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"Interesting  for  you  maybe,  but  a  headache  for  us. 
Plans  for  the  evaluation  had  to  be  changed.   It  seems  that  al- 
though LIP  had  been  granted  a  year's  delay  for  their  field 
trials,  AEP  was  still  expecting  TSA  to  have  all  of  their 
data  to  be  gathered  by  December  1975.   What  had  originally 
been  a  formative  and  summative  evaluation  was  turning  into 
a  formative  one  only.   Something  should  have  been  written 
into  the  contract  to  protect  TSA  in  the  case  of  delays,  but 
it  hadn't  so  we  tried  to  evaluate  what  was  ready. 

Finally  in  early  1974  a  few  teachers  in  two  schools 
began  to  use  the  computer  material  on  a  very  limited  basis. 
This  was  termed  a  "mini-field  trial"  by  LIP.   My  job  became 
one  of  observer.   I  sat  in  the  classrooms  involved  and  took 
copious  notes  on  everything  that  was  happening.   These 
narratives  were  sent  to  TSA  and  a  classroom  observation 
instrument  was  later  drafted  from  them.   I  tested  the 
instrument,  suggested  changes  and  sent  in  more  narratives. 
Things  proceeded  like  that  for  the  rest  of  the  school  year. 

In  mid  1974  a  revised  version  of  the  evaluation  design 
was  presented,  which  was  conceptually  and  technically  well 
thought  out.   An  important  influence  on  the  plan  was  a  con- 
cern for  the  plurality  of  values  exhibited  by  the  various 
audiences.   It  was  recognized  that  no  plan  can  be  equally 
responsive  to  all  audiences.   Nevertheless,  an  attempt  was 
made  to  provide  those  persons,  groups  and  institutions  hav- 
ing an  interest  in  the  outcomes  of  LIP  with  information 
relevant  and  appropriate  to  their  varied  interests.   To  be 
responsive  to  the  goals  and  characteristics  of  the  LIP 
elementary  school  project  itself  required  considerable  custom 
tailoring  of  procedures  and  instruments.   On  the  other  hand 
to  be  responsive  to  the  needs  of  those  outside  the  project 
who  were  interested  in  computer-based  instruction  required 
traditional  measures  and  designs  as  well. 

Demonstrations  of  the  elementary  school  component  were 
to  be  conducted  in  schools  in  Middletown.   Which  schools 
would  participate  was  a  function  of  which  teachers  in  the 
school  districts  volunteered  to  participate  and  the  avail- 
ability of  terminals  and  peripheral  equipment  needed  for 
their  operation.   Thirteen  reading  and  nine  math  teachers 
were  involved  during  the  1974-75  school  year.   (By 
September  1975  ten  new  reading  teachers  and  three  new  math 
teachers  had  joined  the  ranks  of  the  volunteers.) 

The  participants  in  the  evaluation  were  not  limited  to 
those  who  had  volunteered  to  use  LIP  in  their  classrooms. 
It  was  expected  that  comparison  classes  would  be  provided  by 


-35- 


the  school  districts  as  part  of  their  obligation  to  the 
demonstrations." 

"I  notice  that  you  u&ed  the  phha&e  'expected  that  composi- 
tion claAAeA  would  be.  provided.'     Voei>  that  won.d  'expected' 
mean  anything?     Sound*  like  anothen  contractual  problem... 
definition  o\  nx)lei>?" 

"That's  right.   Although  TSA  expected  volunteer 
comparison  classes  it  was  never  in  any  legal  document.   During 
the  1974-1975  school  year  some  comparison  classes  were  obtain- 
ed, but  usually  because  a  school  administrator  had 
'volunteered'  a  teacher  (often  without  that  teacher's  prior 
knowledge) . " 

"But. .  .didn't  you  realize  that  unle*&  people  Mho  wilt 
need  to  buppont  the  evaluation  one  Involved  easily  and  mean- 
ingfully, they  one  not  Likely  to  buppont  it?" 

"We  knew  it.   There  was  just  nothing  that  we  local  people 
could  do,  because  TSA  handled  those  negotiations.   The 
comparison  teachers  resented  every  minute  they  had  to  give 
up  to  our  testing.   This  school  year  the  same  thing  happened. 
TSA  did  not  contact  school  officials  until  the  last  minute 
and  some  teachers  were  not  notified  that  they  would  be 
participating  in  the  evaluation  as  comparison  classrooms 
until  the  week  school  started." 

"1  know  AEF  wanted  the  evaluation  to  bee  li   LIP  had  any 
eiiect  on  the  achievement  and  attitude  in  the  cloAAtioom.     How 
did  TSA  plan  to  go  about  that?" 

"In  the  1974  evaluation  design,  fourteen  key  issues 
were  explicated.   TSA  decided  to  collect  data  relevant  to 
those  issues.   The  sources  and  modes  of  data  acquisition  used 
were  structured  in  terms  of  the  principal  targets  of  the  eval- 
uation:  the  child,  the  classroom,  the  context  of  the  demon- 
stration, and  the  courseware.   The  measures  related  to  the 
child  were  primarily  those  assessing  achievement  and  attitude. 
Student  work  samples  were  also  to  be  included,  as  were  other 
measurement  techniques  such  as  parent  questionnaires  and 
school  cumulative  record  data.   Unfortunately,  when  it  became 
apparent  that  the  1974-75  school  year  would  only  be  a  trial 
run  the  work  sample  and  questionnaires  had  to  be  dropped. 
Resources  had  to  be  spread  over  a  longer  period  of  time  than 
originally  been  planned. 
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In  addition  to  collecting  information  on  the  impact  of 
the  system,  it  was  also  deemed  our  job  to  monitor  the  demon- 
stration in  progress.   Information  concerning  strategies  of 
implementation  was  considered  essential  to  enable  us  to 
identify  classrooms  in  which  the  program  received  a  fair 
trial,  and  to  describe  those  forms  of  implementation  most 
conducive  to  various  outcomes.   A  series  of  classroom 
observations  were  required  to  monitor  this." 

"ObieAvattonA. .  .  I  thought  you'd  neveh.  get  to  them. 
That' *  basically  what  you  wete  kited  to  do,   iAn't  it?    Tkene 
ate  a  jitlian  observation  AckeduleA  around,  did  you  tny 
any  oi  them?" 

"Yes  the  other  local  evaluator  and  I  were  originally 
hired  as  observers.   According  to  the  original  observation 
plan  we  would  receive  an  observation  instrument,  be 
trained  in  its  use  and  go  out  and  observe!!   That  isn't 
quite  how  it  turned  out  though.   As  you  remember,  I 
mentioned  earlier  that  in  the  Spring  of  1974  I  sent  TSA 
many  narratives  which  they  used  to  fashion  an  observation 
scheme.   In  the  Fall  of  1974  the  other  local  evaluator  and 
I  received  copies  of  that  instrument,  which  we  used,  revised, 
used  and  revised  again.   We  trained  ourselves  on  the  job. 
In  addition  to  working  on  the  instrument  for  observing  the 
whole  classroom,  we  also  fashioned  one  for  observing  the 
child  at  the  terminal.   After  almost  a  year's  work,  we 
finally  produced  two  instruments  with  which  we  were  partial- 
ly satisfied. 

The  hardest  part  about  observing  though  is  writing  it 
up.   It  seems  that  no  instrument  says  everything,  so  after 
one  finishes  all  the  checks  or  circles  or  whatever  one  is 
using,  there  is  always  something  important  left  over  to 
write  a  narrative  about." 

"Let' 6  i>ee,   didn' 't  you  mention  bomething  about  on-line 
collection  o{>  data?    That  bound*  tike,  a  good  bounce  oh  in- 
formation.    I've.  keand  that  computer*  can  be  programmed  to 
collect  alt  kind*  oi  thing*.     What  are  you  going  to  do  about 
that?     1  don't  think  you've  mentioned  anything  about  the 
courseware  development  data  eiXker.     Vidn' t  the  evaluation 
have  anything  to  do  wiXh  that  too?" 

"Yes  the  evaluation  was  to  monitor  both  the  on-line 
data  collection  and  the  courseware  development  data.   How- 
ever, neither  of  those  areas  was  under  my  jurisdiction." 
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Recent  federal  legislation  made  it  very  difficult  for  us 
to  obtain  much  of  the  material  which  we  wanted  from  school 
records.   Certain  data  in  school  files  is  now  considered 
confidential  except  to  the  student  or  the  student's  parents. 
This  has  meant  that  though  we  may  obtain  a  general  picture 
of  the  socioeconomic  status  of  the  children  who  are  in  cer- 
tain classrooms  we  may  not  ask  about  each  child  individually, 
unless  we  get  written  permission  from  the  parents  and  this 
type  of  permission  gets  harder  and  harder  to  obtain. 

LIP  was  being  considered  by  the  evaluation  as  constitut- 
ing an  additional  material  resource  to  the  classroom  that  was 
expected  to  have  an  impact  on  how  other  resources  in  the  room 
would  be  perceived  and  used.   The  nature  and  interplay  of 
classroom  resources  was  to  be  monitored  through  several 
procedures,  each  directed  at  selected  aspects  of  the  class- 
room and  using  a  methodology  appropriate  to  its  main  purpose. 
There  were  to  be  six  major  modes  of  data  collection  involved 
in  this  stage:   work  diaries  and  program  summaries,  teacher 
logs,  teacher  interviews  and  questionnaires,  classroom 
observations,  individual  pupil  observations  at  the  terminal 
and  on-line  data  collection." 

"Wo  wondeA  you  be.came.  a  Ah/ieud,   worldly,  -scheming  lady!! 
Anyone  who  could  convince  a  cZaAASioom  tea.cheh.  to  help  wiXh 
those  things  hai>  to  be.  scheming." 

"That  is  true.   Actually  though,  we  did  not  collect  all 
of  that  data.   It  was  just  in  the  evaluation  plan.   When 
resources  were  cut  back  due  to  LIP's  delay,  the  work  diary, 
program  summary  and  teacher  questionnaires  were  lopped  off 
the  plan.   We  did  attempt  the  rest  of  the  techniques.   The 
teacher  log  was  one  which  was  worthwhile  in  some  cases.   The 
teachers  were  asked  to  record  changes  in  the  patterns  of 
their  activities  and  preoccupations,  shifts  in  their 
priorities  and  changes  in  academic  or  social  process  attribu- 
table to  LIP." 

"Those  teacheA  logs  bound  like  a  good  idea,   how  did  they 
work  out?     I'll  bit  some  teachers  did  a  super  job  and  others 
hardly  wrote  anything.     Did  you  get  any  information  that  you 
might  not  otherwise  have  sie.ceA.veJd?    Was>  it  worthwhile?" 

"Yes,  we  did  find  some  interesting  comments.   Our  only 
problem  is  motivating  teachers  to  write  in  the  logs  on  a 
regular  basis.   We  don't  have  time  to  check  up  on  them  so 
we  just  have  to  hope  teachers  are  keeping  up  with  them.   The 
other  technique  which  provided  us  with  interesting  informa- 
tion is  the  teacher  interview.   We  found  teachers  to  be  very 
open  with  us  and  eager  to  provide  us  with  their  views  on  LIP 
and  how  it  effected  their  classroom. 
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"Welt  it  sounds  tike  you  had  enough  to  do  anyway*. 
Let's  get:  back  to  that  teal  wonM.  ofa  evaluation  again.   Jfa 
I  ttemembex  conjtectly ,   you  had  been  working  at  this  job  faon. 
a  yeast  now  and  the,  fteal  faietd  demonstrations  had  not  begun 
yet.     That  was  supposed  to  happen  soon  though  wasn't  it?     Is 
this  when  you  become,  a  fault- faledged  evatuaton.? 

"I'm  not  sure  if  you'd   call   It   full-fledged  or  not,   but 
that  was  when  some  of   the  naiveness,    truthfulness  and  un- 
worldliness  began  to  rub  away.      Even  a  year  after   the 
original   starting  data  the  Computer  Laboratory  people  still 
were  not  ready.      They  did  have   some  classrooms  now  and 
teachers,    but   the  LIP  materials  were  continuing   to  be  develop- 
ed and  modified.      The   terminals  were  not  reliable.      Things 
were  in  a  constant   state  of   flux. 

At   least  we  did   have  some  classrooms   to  observe  and 
with  this  realization  it  became  apparent   that   I  could  not 
cover   21  classrooms  by  myself.      This  was  when  TSA  hired   the 
other  local   evaluator  whom  I've  mentioned  before.      We  tried 
to  observe  and  begin  the  pre-testing  at   the  same  time. 
Obviously  we  were  spreading  ourselves   too  thin,    so  we  hired 
a  local  person  to  do  the  testing   for  us." 

"I'm  way  ahead  ofa  you  thii>  time,     knotheh.  ofa 
Stu.fafale.bejm' s  problems  appealed.     I  can  even  guess. ..it  if, 
stafafaing.     He  says  it  u>  important  to  decide,  who  wilt  have 
overtoil  ftesponsibility  faon.  the  wonk  befaofte  an  evaluation 
begins.     He  also  says  that  questions  like.  ..what  othe/t  Koles 
afte  to  be  manned   [petsonnel?)?     ...what  n.ecn.uitment  ofa 
personnel  wilt  be  done?  ...should  be  answered  ahead  ofa  time. 
Am  I  /tight?" 

"Unfortunately  you  are  right.      From  the  onset  of   the 
evaluation  effort,    no  one  at  TSA  had  overall  responsibility 
for  the  evaluation.      Everyone  who  was   involved   from  Eastern- 
ton  had  this   evaluation  as  only  one  fraction  of  his  work. 
Information  is  promised   to   the   three  of  us  who  work  locally 
but  it  never   comes.      The  three  of   us  who  were  originally 
hired  as  observers   and   testers  have  become  administrators 
and  writers  as  well.      Don't  get  me  wrong,    it's  not  respon- 
sibilities  that  bother  us;    we  rather  enjoy  the  challenges. 
What  does  bother  us   is  that  we  have  had  no  prior   training 
for   the  work  we  are  doing." 

"Anothe/t  problem,   huh?    Stufafalebeam  says  that  ifa  the 
various  persons  afte  to  penfaonm  thevt  nates  efafaectively,  they 
ofaten  need  special  training." 
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"You  are  right.   Training  is  a  key  area  in  evaluation 
work.   Much  valuable  time  has  been  lost  because  training 
became  a  series  of  approximations  until  each  of  us  finally 
learned  the  skills.   Even  more  important  was  the  complete 
lack  of  communication  between  the  Easterton  TSA  group  and 
the  three  of  us.   We  need  feedback  not  only  for  our  own 
information  but  also  to  appear  credible  to  others. 

Enough  of  that. . .another  problem  soon  popped  up.   This 
one  involved  the  scheduling  of  data  collection  activities. 
No  one  at  TSA  had  asked  questions  like:   Is  this  schedule 
reasonable?  Will  the  instruments  be  ready?  Are  their  con- 
flicts between  the  data-gathering  and  other  events  at  the 
school?  How  long  does  it  take  to  administer  the  tests? 
How  many  children  can  take  it  at  once? 

Just  as  important  and  equally  unthought  of  were  questions 
like:   Does  the  budget  reflect  the  evaluation  design?  How  can 
it  be  changed  it  delays  occur?  What  if  new  staff  have  to  be 
hired?  You  can  see  the  way  management  and  economical  prob- 
lems play  a  part  in  the  inefficiency  of  a  project." 

"I  nevex  ticaLLzed  that  evaluating  wcu>  i>o  complicated. 
Whesie.  one.  you  now?  II  I  have,  be.en  ho  Housing  you,  it  Atom* 
tike,  you  one.  in  you/t  tkind  yzasi  on  thii>  job?  lAn' t  evexy- 
tking  i>uppoi>ed  to  be.  whipping  up?  Have  you  had  an  extension? 
Axe  you  itill  collecting  a  lot  o^  data?  Did  you  neaLLze.  how 
much  time,  it  would  take.?  What  do  the  teachex*  think  about 
having  someone  in  theix  claj>6Xoom  testing  all  the  time.?" 

"As  you've  already  guessed  it  took  much  longer  to  ad- 
minister tests  than  anyone  could  have  estimated.   In  most 
classes  testing  conditions  were  bad.   Unless  evaluators  con- 
sider and  respond  to  questions  about  data  gathering  prior  to 
it's  actual  implementation,  there  are  bound  to  be  problems. 
It  is  hard  to  be  unworldly  and  gullible  when  the  world  is 
telephoning  you  every  night  complaining  about  the  tests, 
testers  and  conditions.   A  lot  of  shrewdness  began  to  appear 
about  now;  I  had  to  answer  the  complaints  somehow!" 

"Welt,  what  have,  you  iound  out  about  LIP?    Axe 
thexe  any  xe^>uit6?" 

"Results?  I  wish  I  had  a  lot  to  tell  you,  but  the 
truth  is  that  I  really  don't  know.   The  analyses  performed 
on  last  year's  data  collection  is  done,  but  we  have  not 
received  any  of  the  results.   We  have  received  an  exten- 
sion on  this  year's  collection  and  will  be  able  to  collect 
data  through  May.   However,  with  this  extension  has  come  two 
changes  in  plans  since  August.   We  are  never  sure  when  we 
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are  going  to  get  a  call  from  TSA  saying  that  there  has  been 
another  change  and  that  we  should  scrap  our  schedule  and 
begin  anew." 

"It  it>  obvious  now  why  you  changed  ^fiom  that  ' natheh. 
tAuAting,   naive,   unsophisticated,  gullible  and  unworldly' 
ginZ  to  the  'wofildly,  scheming,  sophisticated,  sh/iewd  and 
experienced'  lady  that  you.  one  now.     The  thing  I  don't 
understand  is  why  do  you  stay  with  it?     Regardless  o^ 
alt  the  changes  which  you  clxuxn  have  taken  place  in  the 
last  two  years  I  i>till  think  you  are  a  pretty  nice  person. 
Tell  me  what' 6  a  nixie  girl  Like  you  doing  in  evaluation?" 

"The  answer  to  that  question  isn't  easy.   Consider  the 
evaluation  plan  I  just  told  you  about.   It  really  had  some 
fine  things  in  it.   The  people  at  TSA  who  wrote  it,  were 
trying  to  find  out  about  the  effectiveness  of  LIP.   Un- 
fortunately, they  did  not  always  run  things  too  well 
administratively.   I  guess  I  stay  in  this  job  because  it  is 
giving  me  a  chance  to  learn  things  that  I  could  never  learn 
from  a  book.   I  stay  in  the  field  of  evaluation  because  I 
believe  that  it  is  a  very  important  one.   Perhaps  the  expe- 
riences that  I  am  having  and  hope  to  continue  to  have  will 
allow  me  to  help  make  things  in  the  field  better.   Evaluation 
is  changing.   Look  at  all  the  new  ideas  which  have  been 
tossed  around  since  Tyler's  model.   It  is  exciting... I 
want  to  be  a  part  of  it." 


"POSTFACE" 

"Well,  that  is  the  story.     It  seems  to  me  that  she 
really  hadn't  changed  that  much.     She  may  be  a  little  wiser, 
but  still  'rather  trusting,   naive,   unsophisticated,  guULible, 
and  unworldly.'     Imagine  thinking  i>he  can  change  evaluation." 
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